How is the border between the Caribbean Sea and Atlantic Ocean defined?

The International Hydrographic Organization came together in 1953 to establish the borders of the oceans and the seas of the world to help guide mariners and to have a standard for consistency. The Organization published a guide from their work entitled "Limits of Oceans and Seas".

As for the border between the Caribbean and the Atlantic the Organization wanted to make sure the Caribbean islands of the lesser Antilles were included with the exception of Barbados which is in the Atlantic. The demarcation line is effectively once the depth gets over 100 fathoms. The line runs from Puerto Rico south to Baya Point in Venezuela.

From Point San Diego (Puerto Rico) Northward along the meridian thereof (65'39' W) to the 100 fathom line, thence Eastward and Southward, in such a manner that all islands, shoals and narrow waters of the Lesser Antilles are included in the Caribbean Sea as far as Galera Point (Northeast extremity of the island of Trinidad). From Galera Point through Trinidad to Galeota Point Southeast extreme and thence to Baa Point 9'32' N, 61' W) in Venezuela.

https://epic.awi.de/29772/1/IHO1953a.pdf (at page 15)