General Scientific Data Formats Reference

This reference covers general-purpose scientific data formats used across multiple disciplines.

Numerical and Array Data

.npy - NumPy Array

Description: Binary NumPy array format Typical Data: N-dimensional arrays of any data type Use Cases: Fast I/O for numerical data, intermediate results Python Libraries:

numpy: np.load('file.npy'), np.save()
Memory-mapped access: np.load('file.npy', mmap_mode='r') EDA Approach:
Array shape and dimensionality
Data type and precision
Statistical summary (mean, std, min, max, percentiles)
Missing or invalid values (NaN, inf)
Memory footprint
Value distribution and histogram
Sparsity analysis
Correlation structure (if 2D)

.npz - Compressed NumPy Archive

Description: Multiple NumPy arrays in one file Typical Data: Collections of related arrays Use Cases: Saving multiple arrays together, compressed storage Python Libraries:

numpy: np.load('file.npz') returns dict-like object
np.savez() or np.savez_compressed() EDA Approach:
List of contained arrays
Individual array analysis
Relationships between arrays
Total file size and compression ratio
Naming conventions
Data consistency checks

.csv - Comma-Separated Values

Description: Plain text tabular data Typical Data: Experimental measurements, results tables Use Cases: Universal data exchange, spreadsheet export Python Libraries:

pandas: pd.read_csv('file.csv')
csv: Built-in module
polars: High-performance CSV reading
numpy: np.loadtxt() or np.genfromtxt() EDA Approach:
Row and column counts
Data type inference
Missing value patterns and frequency
Column statistics (numeric: mean, std; categorical: frequencies)
Outlier detection
Correlation matrix
Duplicate row detection
Header and index validation
Encoding issues detection

.tsv / .tab - Tab-Separated Values

Description: Tab-delimited tabular data Typical Data: Similar to CSV but tab-separated Use Cases: Bioinformatics, text processing output Python Libraries:

pandas: pd.read_csv('file.tsv', sep='\t') EDA Approach:
Same as CSV format
Tab vs space validation
Quote handling

.xlsx / .xls - Excel Spreadsheets

Description: Microsoft Excel binary/XML formats Typical Data: Tabular data with formatting, formulas Use Cases: Lab notebooks, data entry, reports Python Libraries:

pandas: pd.read_excel('file.xlsx')
openpyxl: Full Excel file manipulation
xlrd: Reading .xls (legacy) EDA Approach:
Sheet enumeration and names
Per-sheet data analysis
Formula evaluation
Merged cells handling
Hidden rows/columns
Data validation rules
Named ranges
Formatting-only cells detection

.json - JavaScript Object Notation

Description: Hierarchical text data format Typical Data: Nested data structures, metadata Use Cases: API responses, configuration, results Python Libraries:

json: Built-in module
pandas: pd.read_json()
ujson: Faster JSON parsing EDA Approach:
Schema inference
Nesting depth
Key-value distribution
Array lengths
Data type consistency
Missing keys
Duplicate detection
Size and complexity metrics

.xml - Extensible Markup Language

Description: Hierarchical markup format Typical Data: Structured data with metadata Use Cases: Standards-based data exchange, APIs Python Libraries:

lxml: lxml.etree.parse()
xml.etree.ElementTree: Built-in XML
xmltodict: Convert XML to dict EDA Approach:
Schema/DTD validation
Element hierarchy and depth
Namespace handling
Attribute vs element content
CDATA sections
Text content extraction
Sibling and child counts

.yaml / .yml - YAML

Description: Human-readable data serialization Typical Data: Configuration, metadata, parameters Use Cases: Experiment configurations, pipelines Python Libraries:

yaml: yaml.safe_load() or yaml.load()
ruamel.yaml: YAML 1.2 support EDA Approach:
Configuration structure
Data type handling
List and dict depth
Anchor and alias usage
Multi-document files
Comments preservation
Validation against schema

.toml - TOML Configuration

Description: Configuration file format Typical Data: Settings, parameters Use Cases: Python package configuration, settings Python Libraries:

tomli / tomllib: TOML reading (tomllib in Python 3.11+)
toml: Reading and writing EDA Approach:
Section structure
Key-value pairs
Data type inference
Nested table validation
Required vs optional fields

.ini - INI Configuration

Description: Simple configuration format Typical Data: Application settings Use Cases: Legacy configurations, simple settings Python Libraries:

configparser: Built-in INI parser EDA Approach:
Section enumeration
Key-value extraction
Type conversion
Comment handling
Case sensitivity

Binary and Compressed Data

.hdf5 / .h5 - Hierarchical Data Format 5

Description: Container for large scientific datasets Typical Data: Multi-dimensional arrays, metadata, groups Use Cases: Large datasets, multi-modal data, parallel I/O Python Libraries:

h5py: h5py.File('file.h5', 'r')
pytables: Advanced HDF5 interface
pandas: HDF5 storage via HDFStore EDA Approach:
Group and dataset hierarchy
Dataset shapes and dtypes
Attributes and metadata
Compression and chunking strategy
Memory-efficient sampling
Dataset relationships
File size and efficiency
Access patterns optimization

.zarr - Chunked Array Storage

Description: Cloud-optimized chunked arrays Typical Data: Large N-dimensional arrays Use Cases: Cloud storage, parallel computing, streaming Python Libraries:

zarr: zarr.open('file.zarr')
xarray: Zarr backend support EDA Approach:
Array metadata and dimensions
Chunk size optimization
Compression codec and ratio
Synchronizer and store type
Multi-scale hierarchies
Parallel access performance
Attribute metadata

.gz / .gzip - Gzip Compressed

Description: Compressed data files Typical Data: Any compressed text or binary Use Cases: Compression for storage/transfer Python Libraries:

gzip: Built-in gzip module
pandas: Automatic gzip handling in read functions EDA Approach:
Compression ratio
Original file type detection
Decompression validation
Header information
Multi-member archives

.bz2 - Bzip2 Compressed

Description: Bzip2 compression Typical Data: Highly compressed files Use Cases: Better compression than gzip Python Libraries:

bz2: Built-in bz2 module
Automatic handling in pandas EDA Approach:
Compression efficiency
Decompression time
Content validation

.zip - ZIP Archive

Description: Archive with multiple files Typical Data: Collections of files Use Cases: File distribution, archiving Python Libraries:

zipfile: Built-in ZIP support
pandas: Can read zipped CSVs EDA Approach:
Archive member listing
Compression method per file
Total vs compressed size
Directory structure
File type distribution
Extraction validation

.tar / .tar.gz - TAR Archive

Description: Unix tape archive Typical Data: Multiple files and directories Use Cases: Software distribution, backups Python Libraries:

tarfile: Built-in TAR support EDA Approach:
Member file listing
Compression (if .tar.gz, .tar.bz2)
Directory structure
Permissions preservation
Extraction testing

Time Series and Waveform Data

.wav - Waveform Audio

Description: Audio waveform data Typical Data: Acoustic signals, audio recordings Use Cases: Acoustic analysis, ultrasound, signal processing Python Libraries:

scipy.io.wavfile: scipy.io.wavfile.read()
wave: Built-in module
soundfile: Enhanced audio I/O EDA Approach:
Sample rate and duration
Bit depth and channels
Amplitude distribution
Spectral analysis (FFT)
Signal-to-noise ratio
Clipping detection
Frequency content

.mat - MATLAB Data

Description: MATLAB workspace variables Typical Data: Arrays, structures, cells Use Cases: MATLAB-Python interoperability Python Libraries:

scipy.io: scipy.io.loadmat()
h5py: For MATLAB v7.3 files (HDF5-based)
mat73: Pure Python for v7.3 EDA Approach:
Variable names and types
Array dimensions
Structure field exploration
Cell array handling
Sparse matrix detection
MATLAB version compatibility
Metadata extraction

.edf - European Data Format

Description: Time series data (especially medical) Typical Data: EEG, physiological signals Use Cases: Medical signal storage Python Libraries:

pyedflib: EDF/EDF+ reading and writing
mne: Neurophysiology data (supports EDF) EDA Approach:
Signal count and names
Sampling frequencies
Signal ranges and units
Recording duration
Annotation events
Data quality (saturation, noise)
Patient/study information

.csv (Time Series)

Description: CSV with timestamp column Typical Data: Time-indexed measurements Use Cases: Sensor data, monitoring, experiments Python Libraries:

pandas: pd.read_csv() with parse_dates EDA Approach:
Temporal range and resolution
Sampling regularity
Missing time points
Trend and seasonality
Stationarity tests
Autocorrelation
Anomaly detection

Geospatial and Environmental Data

.shp - Shapefile

Description: Geospatial vector data Typical Data: Geographic features (points, lines, polygons) Use Cases: GIS analysis, spatial data Python Libraries:

geopandas: gpd.read_file('file.shp')
fiona: Lower-level shapefile access
pyshp: Pure Python shapefile reader EDA Approach:
Geometry type and count
Coordinate reference system
Bounding box
Attribute table analysis
Geometry validity
Spatial distribution
Multi-part features
Associated files (.shx, .dbf, .prj)

.geojson - GeoJSON

Description: JSON format for geographic data Typical Data: Features with geometry and properties Use Cases: Web mapping, spatial analysis Python Libraries:

geopandas: Native GeoJSON support
json: Parse as JSON then process EDA Approach:
Feature count and types
CRS specification
Bounding box calculation
Property schema
Geometry complexity
Nesting structure

.tif / .tiff (Geospatial)

Description: GeoTIFF with spatial reference Typical Data: Satellite imagery, DEMs, rasters Use Cases: Remote sensing, terrain analysis Python Libraries:

rasterio: rasterio.open('file.tif')
gdal: Geospatial Data Abstraction Library
xarray with rioxarray: N-D geospatial arrays EDA Approach:
Raster dimensions and resolution
Band count and descriptions
Coordinate reference system
Geotransform parameters
NoData value handling
Pixel value distribution
Histogram analysis
Overviews and pyramids

.nc / .netcdf - Network Common Data Form

Description: Self-describing array-based data Typical Data: Climate, atmospheric, oceanographic data Use Cases: Scientific datasets, model output Python Libraries:

netCDF4: netCDF4.Dataset('file.nc')
xarray: xr.open_dataset('file.nc') EDA Approach:
Variable enumeration
Dimension analysis
Time series properties
Spatial coverage
Attribute metadata (CF conventions)
Coordinate systems
Chunking and compression
Data quality flags

.grib / .grib2 - Gridded Binary

Description: Meteorological data format Typical Data: Weather forecasts, climate data Use Cases: Numerical weather prediction Python Libraries:

pygrib: GRIB file reading
xarray with cfgrib: GRIB to xarray EDA Approach:
Message inventory
Parameter and level types
Spatial grid specification
Temporal coverage
Ensemble members
Forecast vs analysis
Data packing and precision

.hdf4 - HDF4 Format

Description: Older HDF format Typical Data: NASA Earth Science data Use Cases: Satellite data (MODIS, etc.) Python Libraries:

pyhdf: HDF4 access
gdal: Can read HDF4 EDA Approach:
Scientific dataset listing
Vdata and attributes
Dimension scales
Metadata extraction
Quality flags
Conversion to HDF5 or NetCDF

Specialized Scientific Formats

.fits - Flexible Image Transport System

Description: Astronomy data format Typical Data: Images, tables, spectra from telescopes Use Cases: Astronomical observations Python Libraries:

astropy.io.fits: fits.open('file.fits')
fitsio: Alternative FITS library EDA Approach:
HDU (Header Data Unit) structure
Image dimensions and WCS
Header keyword analysis
Table column descriptions
Data type and scaling
FITS convention compliance
Checksum validation

.asdf - Advanced Scientific Data Format

Description: Next-gen data format for astronomy Typical Data: Complex hierarchical scientific data Use Cases: James Webb Space Telescope data Python Libraries:

asdf: asdf.open('file.asdf') EDA Approach:
Tree structure exploration
Schema validation
Internal vs external arrays
Compression methods
YAML metadata
Version compatibility

.root - ROOT Data Format

Description: CERN ROOT framework format Typical Data: High-energy physics data Use Cases: Particle physics experiments Python Libraries:

uproot: Pure Python ROOT reading
ROOT: Official PyROOT bindings EDA Approach:
TTree structure
Branch types and entries
Histogram inventory
Event loop statistics
File compression
Split level analysis

.txt - Plain Text Data

Description: Generic text-based data Typical Data: Tab/space-delimited, custom formats Use Cases: Simple data exchange, logs Python Libraries:

pandas: pd.read_csv() with custom delimiters
numpy: np.loadtxt(), np.genfromtxt()
Built-in file reading EDA Approach:
Format detection (delimiter, header)
Data type inference
Comment line handling
Missing value codes
Column alignment
Encoding detection

.dat - Generic Data File

Description: Binary or text data Typical Data: Instrument output, custom formats Use Cases: Various scientific instruments Python Libraries:

Format-specific: requires knowledge of structure
numpy: np.fromfile() for binary
struct: Parse binary structures EDA Approach:
Binary vs text determination
Header detection
Record structure inference
Endianness
Data type patterns
Validation with documentation

.log - Log Files

Description: Text logs from software/instruments Typical Data: Timestamped events, messages Use Cases: Troubleshooting, experiment tracking Python Libraries:

Built-in file reading
pandas: Structured log parsing
Regular expressions for parsing EDA Approach:
Log level distribution
Timestamp parsing
Error and warning frequency
Event sequencing
Pattern recognition
Anomaly detection
Session boundaries