Here are a few ideas I got:

1. Look up the ship type:

As long as this ship you are seeing is not made by a to date unknown civilication, your databases should be able to identify the ship type, and show you some basic information for that.

2. Estimate by size:

As long as you can measure the distance, you can measure the size of such a ship. Now, your engineers will have an estimate for how much space is taken up by machinery, how much mass is used for the hull, how much space is needed for crew and so on. They know what materials are used commonly, and by multiplying the estimated volume with an average density you can estimate the mass of the ship. It is not a precise estimate, but your engineers should be pretty near to the exact value.

3. Measure the acceleration and energy output of the engines:

Every engine known to us emits radiation in some form due to conversion losses. As long as you can measure the acceleration of such a ship, the radiation of the engines and can identify the type of engine, you should be able to get a good estimate how much energy is used to power the engine, how much force the engines generate and in turn how much mass the ship has.

Everything you need for the above methods are good optical sensors and radiation sensors.

The vacuum of space act as a dielectric.

If you are close enough, provide a net electric charge to your own ship. This will induce a charge also on the probed ship, and will trigger an electrostatic attraction between the two.

Measure your velocity with respect to background and your velocity with respect to the probed ship.

Since you know the charge involved and you can determine the resulting force exerted on the probed ship. The resulting force will change your velocity and the one of the probed ship. You know your mass and your velocity both with respect to background and to probed ship, the only unknown parameter is the probed ship mass.

In Civilized Areas, They will Tell You Automatically

Today, ships and aircraft have systems that self report their name, size, exact location, velocity, destination etc. For ships, this is AIS, and for aircraft it is ADS-B. The systems automatically squawk their info at regular intervals, so as soon as you are in receiving range you know what the contact picture is.

It is often required by the relevant authorities, as it significantly improves safety when everyone knows where everyone else is.

Otherwise, IR will give a Good Guess

All vessels will have a power source. This power source is going to have waste heat, which will radiate from the vessel evenly. (Unless the owner pumps air/fluids around to deliberately alter the heat transfer.) This waste heat will be subject to one over r squared losses. If you can determine a range, you can determine how much this blackbody radiation will have dropped off, and you can get an estimate of the size of the power source.

Power is going to be strongly related to vessel size and acceleration. The exact equation is probably going to be complex, but luckily the universe is full of vessels providing you with empirical data!

Muography.

https://en.wikipedia.org/wiki/Muography

Muography is an imaging technique that produces a projectional image
of a target volume by recording elementary particles, called muons,
either electronically or chemically with materials that are sensitive
to charged particles such as nuclear emulsions. Cosmic rays from outer
space generate muons in the Earth’s atmosphere as a result of nuclear
reactions between primary cosmic rays and atmospheric nuclei. They are
highly penetrative and millions of muons pass through our bodies every
day.

Muons traverse a solid object according the the mass of the object, more dense objects stopping muons more than less dense objects. One can use ambient muons to produce images. This is how the interior of a volcano was imaged. NASA has a proposal to use ambient muons to image the interior of asteroids in space.

https://www.nasa.gov/content/deep-mapping-of-small-solar-system-bodies-with-galactic-cosmic-ray-secondary-particle-1

Muography has been used to characterize very dense objects. If you are digging it, read this whole paper where they characterized old objects from nuclear waste sites. https://royalsocietypublishing.org/doi/pdf/10.1098/rsta.2018.0048. Lots of images of stuff with stuff inside! Here is an image of a contaminated crucible encased in glass, minus the glass and then minus the crucible, with only the radioactive contaminants.

Muography can let you image a ship passively, without it knowing it is being imaged. That might have its uses.

Backscatter muography is near future scifi - the idea would be to have your own muon source, like your own light source and actively image something by shooting muons at it and then capturing those which are reflected. Just as light things reflect visible light more, dense materials will reflect muons more, and just as one characterizes density passively by capturing wild muons which traverse from the far side of the object heading towards you, you could actively image by spraying your artificial muons at the object and characterizing scatter back towards you.

Probe indirectly

Launch a small probe of precisely known mass so that it passes at a known distance from the ship. The probe might be as opaque and undetectable as could be. At a predetermined moment the probe shoots a low-power laser signal to the mothership, allowing its position to be precisely determined. From there, you can work back the deflection on the trajectory caused by the gravitational mass of the unknown ship.

Probe (almost) directly

Send two probes at different distances from the unknown ship, depending on how sensitive their gravitational sensors are. Measure acceleration on both probes. These will be caused by all nearby masses (the mothership included), but using two probes, the residual that has quadratic dependency on the unknown ship's distance once the other masses are ruled out will give the unknown ship's mass.

For example if all other masses are sufficiently far away, the distance between them and probes P and Q may be taken as constant, so the differential will be zero. The acceleration on probe P will then be given by G(M/a^2 - X/(b+d)^2) while that on probe Q by G(M/(a+d)^2 - X/b^2), with M and X the masses of the mothership and the unknown's, a and b the distances of the first probe from the mothership M and the unknown, and d the distance between the probes:

 M P Q X
 |--- a ------|--- d ---|--------------------- b ----------------|

Assuming b, especially, is known with sufficient accuracy (phased laser ranging, maybe?) and gravimeters (not accelerometers, thanks @wizzwizz) of sufficient precision are available, the two probes could even be mounted on a fixed "antenna" (it is not necessary for M, P, Q and X to be aligned on a line, but this simplifies things).

Shine a laser at it, and measure the (exceedingly slight) change in momentum.

Several years later, in 1922, physicist Arthur Compton performed an
experiment which led to the discovery of the Compton Effect. Proving
Einstein correct, Compton showed that photons indeed have momentum
which is transferrable to materials that have a mass. Compton was
awarded the 1927 Noble Prize in Physics for demonstrating that photons
can transfer their momentum to the electrons with which they collide
inside an atom.

I am, of course, assuming that by the time there are enough spaceships out there that one needs to worry about determining their mass, we will have the technology to be able to detect very minute changes in momentum that a laser beam would impart on a ship. Yes, I am aware that it would be easier to detect the change in momentum caused by a flea smacking into a super jumbo jet, but hey, we are talking about the future here.

Equipment that can survey the gravitational potential on the order of microgals is quite common today in geological industry and academia, known as gravimetry.

The machines used as instrumentation, gravimeters, are quite interesting, but ultimately they are able to derive densities by way of directly measuring differing gravitational acceleration at many points. These have even been used to probe non-geological manmade-scale objects e.g. when French scientists surveyed the Cheops pyramid for unknown chambers.

A tech advanced gravimeter (perhaps using an array of atom chip nano gravimeters) paired with a good gravimetric map of the navigation area should allow measurement of the mass of an anomalous body. With enough resolution you could even possibly map the density distribution of the vessel and track masses moving on the interior.

Scan them. Fire a very large burst of neutrinos at them and measure how much gets reflected back. In general the more massive an object is the more return signal you will get. It works okay for an estimated order of magnitude.

Unlike other answers this probably won't be recognized as an act of violence, doesn't disturb their trajectory and gives you an answer near to light speeds.

Edit: Actually, I'm no physicist but I think you won't get neutrinos back, you might get electrons, but I'm just not knowledgeable enough to say for sure.

Shoot them with a powerful laser and see how much the momentum of their ship changes.

You didn't mention anything about leaving the ship intact.

A patient passive approach could work. Observe ship apparent size and structure (visual angle) relative to your ship. Use an optical distance range finder. Integrate the number of naturally present cosmic rays passing through the other ship into your ship in order to passively estimate density - The ice cube observatory uses similar methods and works for space ships up to almost the thickness of Earth. Get an estimate of the other ships surface area & unseen sides by capacitance (just measure your own ships capacitance, deviation from typical solo deep space value indicates parallel capacitance of the nearby ship). Calculate surface to volume ratio and add density estimate from cosmic ray result for an estimate of mass that improves with more cosmic ray integrations.

Gravitational Field, nearly exactly how we are measuring EM field currently. LIGO proved it can be done for large object in 2017 when 2 black holes collided and generated gravitational waves.

Starship in the future should be in the scale of million - trillion tonnes and moving at at least 10-20% speed of light, this would make considerable 'huge' amount of gravitational waves if moving 'close' together (by close I mean a few thousands kilometers). Heck we can measure gravitational waves emitted by black holes light years away now, absolutely we can do that for smaller objects within a few thousands kilometers in 200-300 years, especially if that object moves close to speed of light since it generates more identifiable ripples.

I'm pretty sure we can work the math out to the exact dimensions of LIGO-like version needed to do this, but it would be terrifyingly difficult for people like us. I have strong background in electric engineering and still cannot grasp most part of it.

You didn't say how long you want to take to figure out the other ship's mass. If you aren't in a hurry and it's just you and the other ship with no other ships or masses nearby, and (ideally) neither of you is accelerating, then you can simple take periodic measurements of the distance between your ship and theirs. Two distance measurements taken close together can tell you your current speed. Then at longer intervals take more measurements and determine how much your distance differs from expectations. The difference should be mainly due to gravitational attraction between ships.

Note that this method works best if the other ship is stationary relative to you, isn't terribly far away, and remains this way for a few weeks, maybe less depending on how accurately you can measure the distance between your ship and theirs using radar or some other mechanism. This situation happened in "2001: A Space Odyssey" where during the trip to Jupiter there were two ships near each other and they had to make minor course corrections during the trip due to their mutual gravitational attraction.

Grab them with a tractor beam (or cable with a magnet at its end if you're inclined to be low tech and their hull is magnetic) and try to tug it.

Provided you have some way of precise spatial reference (launching several small drones in at least 3 directions should suffice as low tech solution), and you knowing your own mass and power of your engines, you should be able to calculate inertia of additional mass you're trying to move.

Assuming no magic technology to provide the answer, and that I was writing a hard scifi story where this knowledge was important, then ... as has been said above 'Newton is your Daddy!'

The same method used to measure masses of all the planets in our solar system would work in this case. To summarize my interpretation of the original technique is by iterating through assumed mass values and computing the paths followed by the planets in their orbits to predict where the planet will be in the future. By comparing the actual paths to the predicted paths, the error in the guesstimate for the planetary masses can be improved.

It's my understanding that this is how early astronomers came up with the values for planetary masses, and I also believe that there are slight wobbles in planetary orbits that our guessimates can't model; suggesting that there are more masses to be accounted for in our equations. But, our models are good enough to launch space probes past Pluto, so they are good enough for now.

For spaceships, the idea would be similar. Spaceship A wants to travel along a specific path in 4D (x,y,z,t). It knows its own mass very accurately. And can measure the distance to any object it can see via lidar, radar, optical means, etc. The spaceship computes the thrust vector to move it along the path, any errors in the actual path represent either unidentified masses or inaccuracies in mass guesstimates.
Iterating over the computations until the errors in navigating go to zero, or near zero, then you know the mass of everything around you.

Similarly, if the vector required to keep a ship moving along a specific path suddenly changed or slowly changed, then that would indicate something moving near or away from your ship. This would provide a means for detecting hidden ships.

The math for this kind of stuff is very common in scientific applications, radar systems, sonar systems, and cell phone technology -- albeit not computing errors in mass, but for figuring out parameters that affect the transmission, reception, or interpretation of data.

And since gravitatal forces are limited by relativistic limits there is a practical limit to the range that objects could be detected and 'measured.'

Less than a hundreds years ago, we didn't have any application for electromagnetic waves, nor didn't know how to sense it. Now we can detect gravitational waves (well, only the strong). In future, we may have miniaturized sensitive technology to measure the disturbance in space-time.
Who knows, may be we can sense the mass of the ship by measuring the turbulence it causes in the space-time.

Late to the party, but...

Measure the tidal effect of their gravitational well.

Using a network of extremely accurate force measurement devices (direction and magnitude) you could get a map of the local gravitational topography. Subtract your ship's baseline to remove its own gravitational field from the data and you'll get a very small tidal effect on a line that points to the target ship. Detectors on the side closest to the target will be trying to accelerate towards it faster than detectors on the other side of your ship by a tiny amount that is can be used to calculate the mass of the target.

Of course we're talking about measuring exceptionally small values - on the order of 67 nano Newtons of tidal force over 500m with a target mass of 1 gigaton. You'd best keep your crew from breathing too hard while all this is being measured.

The Newton method

Have a large array of highly sensitive movable ball-bearings (similar types of lumps of matter), perhaps mounted on precisely calibrated springs or something, suspended in a vacuum.

The ball-bearings will be pulled towards the target object with a force proportional to its mass, which can be calculated by how much the ball-bearings move.

Of course, you'll have to take into account all the objects in the area, including your own ship and the people and objects therein (including the other ball-bearings and springs) which is why you need such a large array of sensors - each will be moved by a different amount depending on its distance from each object, allowing an AI to use the movement of each ball-bearing to isolate object.

The algorithm to do that is left as an exercise to the reader :-p

You can't measure it directly at a distance with current technology.

Tech however at a given time is fairly well constrained. E.g. If you are building battleships, then knowing the dimensions of the battleship give you a pretty good idea of the mass. Space allows extreme magnification of images, so getting an approximate mass at a distance of several AU's is reasonable.

Another factor is the power/mass ratio. This will affect acceleration, and possibly also weaponry. A destroyer has a much higher power/weight ratio than a battleship. Thinner armour, smaller guns. Destroyers run rings around fleets and convoys to intercept subs.

In space you also look at the waste heat. The volume of a ship goes up with the cube of linear dimensions, while the surface area goes up only as the square. A big ship has a bigger problem getting rid of waste heat. It's radiators will run at higher temps.

You would also monitor communications. Difficult. Comm channels are likely to be tightly beamed.

All of these are subject to a certain amount of spoofing:

• A small ship can inflate a large metallic balloon and appear to be much larger.


• A small ship, by manouvering at well below max acceleration may be thought to be larger.


• A small ship, by directing it's cooling heat at you can appear to be disposing of more heat -- signature of a larger ship.


• A large ship, by directing waste heat away from you appears smaller


• A large ship, if shaped like a disk or a brick can maintain an orientation edge on or end on and appear smaller.


• A large ship, with the outline of a small ship in white and the rest of the ship black, may be classed as a small ship at long range. Of course black doesn't help the heat disposal problem.

Detectors: Visual gives you the most information about the detail. Far infrared gives you more information about the skin temperature, and hence the power use. The outline technique above wouldn't fool an IR scan.

Having scouts spread throughout the space of interest makes deception harder. A small detector, disguised or part of an asteroid would be nearly impossible to spot. If space also had several radar illuminator ships, the passive scouts would get radar distances -- You get both the reflection of the radar off the enemy, as well as the time signal from the radar ship. That and geometry give you a solution of the distance.

Note that as soon as you change technology, all this goes out the window. E.g. at present one of the ways we would measure is the amount of rocket exhaust and the resulting acceleration. Replace that with ion drives, and we may not even see the exhaust.

Analogues:

Submarines are identified by their sound signatures. And some of the same forms of deception are used, pretending to be something they are not. (Usually however a sub's goal is to be undetected.

Radar can determine the difference between a Cessna and a Boeing by strength of reflection, as well as the way it manoevers. Adding a corner reflector increases the radar return substantially making the Cessna look bigger. Stealth coatings on fighters make the radar return vanishingly small.

In the days of sail, a ship of the line could be spotted further away -- taller masts. At closer range, you could look at the rigging and get a good idea of the size of the ship. Counting gun decks was another way.

The uncertainties add to the fun in your universe. Allows for possible ruses des guerre