Handling Dates and Times in Pandas Efficiently

Datetime operations can be slow if done wrong. Here's how to handle dates efficiently in Pandas:

1. Parse Dates When Loading Data

# Slow: Parse after loading ❌
df = pd.read_csv('data.csv')
df['date'] = pd.to_datetime(df['date'])

# Fast: Parse during loading ✅
df = pd.read_csv('data.csv', parse_dates=['date', 'created_at'])

2. Specify Date Format

# Slow: Pandas guesses format (tries many patterns) ❌
df['date'] = pd.to_datetime(df['date'])

# Fast: Tell Pandas the format ✅
df['date'] = pd.to_datetime(df['date'], format='%Y-%m-%d')
# Up to 10x faster on large datasets!

Common Format Codes:

• %Y-%m-%d → 2025-01-15
• %m/%d/%Y → 01/15/2025
• %Y-%m-%d %H:%M:%S → 2025-01-15 14:30:00
• %d-%b-%Y → 15-Jan-2025

3. Extract Date Components Efficiently

# Using .dt accessor (vectorized)
df['year'] = df['date'].dt.year
df['month'] = df['date'].dt.month
df['day'] = df['date'].dt.day
df['day_of_week'] = df['date'].dt.dayofweek  # 0=Monday
df['day_name'] = df['date'].dt.day_name()     # 'Monday'
df['quarter'] = df['date'].dt.quarter

4. Filter by Date Ranges

# Filter by date
df[df['date'] > '2025-01-01']
df[df['date'].between('2025-01-01', '2025-12-31')]

# Filter by year/month (efficient!)
df[df['date'].dt.year == 2025]
df[df['date'].dt.month == 1]

# Last N days
from datetime import datetime, timedelta
last_30_days = datetime.now() - timedelta(days=30)
df[df['date'] > last_30_days]

5. Resampling Time Series

# Aggregate by time periods
df.set_index('date').resample('D').sum()   # Daily
df.set_index('date').resample('W').mean()  # Weekly
df.set_index('date').resample('M').sum()   # Monthly
df.set_index('date').resample('Q').sum()   # Quarterly

6. Time Differences

# Calculate time differences
df['days_since_signup'] = (pd.Timestamp.now() - df['signup_date']).dt.days
df['hours_between'] = (df['end_time'] - df['start_time']).dt.total_seconds() / 3600

# Age calculation
from datetime import datetime
df['age'] = (datetime.now() - df['birth_date']).dt.days / 365.25

7. Handle Timezones

# Add timezone
df['date'] = df['date'].dt.tz_localize('UTC')

# Convert timezone
df['date_est'] = df['date'].dt.tz_convert('America/New_York')

# Remove timezone (make naive)
df['date'] = df['date'].dt.tz_localize(None)

Performance Tips

• ✅ Always specify format in pd.to_datetime()
• ✅ Use parse_dates when reading CSVs
• ✅ Use .dt accessor instead of .apply(lambda)
• ✅ Set datetime column as index for time series operations
• ❌ Don't use Python datetime operations in loops