Filtering Data

Filtering Data#

In geospatial analysis, filtering vector data is a common task to refine datasets based on specific conditions. Kalpa offers a powerful and flexible filtering mechanism using Python’s Pandas-style operations. This chapter demonstrates how to filter vector data using conditions on one or multiple columns, enabling you to create tailored datasets for your analysis.

Vector Filtering#

The VectorLayerFiltering function allows you to filter a vector layer or a file containing vector data (.gpkg/.shp) using custom conditions written in Python’s Pandas-style operations.

Function Overview#

The VectorLayerFiltering function applies a condition to filter rows in a vector dataset and returns the filtered result.

Key Arguments:

  • A file path to a vector dataset or selection of a layer (e.g., .gpkg, .shp).

  • filter_condition: A Python condition for filtering rows. The condition uses the format:

    row['column_name'] <condition> value

    You can combine multiple conditions using logical operators like and, or, and not.

Examples#

Single-Column Based Filtering#

  • Scenario: Filter all rows where the Population column is greater than 1,000.

  • Condition String: row['Population'] > 1000

Multi-Column Based Filtering#

  • Scenario: Filter rows where the Population is greater than 1,000 and the City starts with the letter ‘C’.

  • Condition String: row['Population'] > 1000 and row['City'].startswith('C')

Filtering Using Equality Conditions#

  • Scenario: Filter rows where the City column is equal to ‘B’.

  • Condition String: row['City'] == 'B'

Combining Conditions with OR#

  • Scenario: Filter rows where the Population is less than 1,000 or the City starts with ‘A’.

  • Condition String: row['Population'] < 1000 or row['City'].startswith('A')

Filtering by Distance (Geospatial Attributes)#

  • Scenario: Filter rows where the distance to a fault line is less than 5 km.

  • Condition String: row['Fault_Dist'] < 5

Filtering Rows with Numerical Ranges#

  • Scenario: Filter rows where Population is between 500 and 1,500.

  • Condition String: 500 <= row['Population'] <= 1500

Filtering Rows Based on String Patterns#

  • Scenario: Filter rows where the City name contains the letter ‘a’ (case-insensitive).

  • Condition String: row['City'].str.contains('a', case=False)

Filtering Rows with Missing or Null Values#

  • Scenario: Filter rows where the geometry column is None (missing).

  • Condition String: row['geometry'] is None

Filtering Rows Based on Multiple Conditions (Advanced)#

  • Scenario: Filter rows where Population is greater than 1,000, and the City does not start with ‘A’.

  • Condition String: row['Population'] > 1000 and not row['City'].startswith('A')

Filtering Rows Using Custom Functions#

  • Scenario: Use a custom function to filter rows where the City name length is greater than 1 character.

  • Condition String: len(row['City']) > 1

Filtering Geospatial Data by Attribute and Proximity#

  • Scenario: Filter rows where faults are older than 50 million years and within 10 km of the grid points.

  • Condition String: row['Fault_Age'] > 50 and row['Fault_Dist'] < 10

Filtering Using Logical OR Conditions#

  • Scenario: Filter rows where the City is either ‘A’ or ‘C’.

  • Condition String: row['City'] in ['A', 'C']

Filtering by Area or Length Attributes#

For vector datasets with polygons or lines, you can filter by geometric properties such as area or length.

  • Scenario: Filter polygons where the area is greater than 1,000 square meters. - Condition String: row['geometry'].area > 1000

  • Scenario: Filter line features where the length is less than 500 meters. - Condition String: row['geometry'].length < 500

Tips for Writing Filtering Conditions#

  1. Use Logical Operators: Combine conditions with and, or, or not to create complex queries.

  2. Check Data Types: Ensure your column data types match the condition. Numeric values should not be compared to strings.

  3. Handle Missing Values: Use Pandas-style operations like row['column'].notnull() to filter out rows with missing data.

  4. Validate Columns: Ensure the columns used in filtering exist in the dataset.

  5. Optimize Conditions: Use simple, efficient conditions to avoid unnecessary computational overhead.

  6. Test Conditions: Before applying complex filters, test them on a small subset of data to ensure correctness.

  7. Export Results: Save filtered datasets for further analysis or visualization.