We compute the Euler characteristic of the structure sheaf of the Brill-Noether locus of linear series with special vanishing at up to two marked points. When the Brill-Noether number rho is zero, we recover the Castelnuovo formula for the number of special linear series on a general curve; when rho = 1, we recover the formulas of Eisenbud-Harris, Pirola, and Chan-Martin-Pflueger-Teixidor for the arithmetic genus of a Brill-Noether curve of special divisors. These computations are obtained as applications of a new determinantal formula for the K-theory class of certain degeneracy loci. Our degeneracy locus formula also specializes to new determinantal expressions for the double Grothendieck polynomials corresponding to 321-avoiding permutations and gives double versions of the flagged skew Grothendieck polynomials recently introduced by Matsumura. Our result extends the formula of Billey-Jockusch-Stanley expressing Schubert polynomials for 321-avoiding permutations as generating functions for flagged skew tableaux.