You can cut off a chunk and send it off to a lab to test for asbestos content-- obviously, don't do that in an obvious place if you're not immediately planning on replacing it or covering it. If you can lift a baseboard and get material from the edge, then cover the missing section back over with the baseboard, that's often a good location. For the mastic, you can get it wet and scrape up some and send it off for testing. The lab will almost always have specific instructions on safe ways to collect material for testing and how much you need. Similar appearing materials are all over the place on actual asbestos content.
Vinyl asbestos tiles tend to be square and have kind of a marbled look. They are often, but not always, 9x9. The vinyl asbestos tiles are generally quite low risk compared to the asbestos-backed sheet vinyl, because the asbestos fibers are bound in a matrix with the vinyl. Unless you pulverize them in removal, you're not going to get airborne fibers taking them out. If you contrast whatever your local recommendations are for VAT removal with sheet vinyl removal, they're usually pretty different-- the main thing people are concerned about with VAT is how you dispose of it. For sheet vinyl, though, they're really worried about how you take it up.
Mastics can be pretty difficult to get up. For example, the black mastic was really easy to get up with Bean-E-Doo (whatever they call it now)-- anything else, though, forget it. The remnants of the Bean-E-Doo itself were challenging (lots of soap, water, scrubbing, etc.) However, though the same chemical said it works on old carpet adhesives, it didn't work at all for me. If you have things tested and the tile's asbestos and the mastic's not, the tiles are pretty easy to remove nondestructively with steam (e.g., a steam iron or a wallpaper steamer, or you can rent a big machine that heats the tiles for you). Then you can rent a big commercial scraper to get the mastic up, which is the most efficient. If the mastic does contain asbestos, either it shifts easily with heat or it doesn't, it's water soluble or oil soluble, etc. Use whatever people think is the best method for the specific type of mastic you have. Or paint over the mastic with one of the sealing compounds if the standard removal methods don't work. Since your tiles are loose, though, I'd take them out before putting any new flooring down.
Q