To upload data to StreamPULSE, you must be a registered user (register here). You can then upload sensor data, which are necessary for accurately modeling metabolism, and manually collected data, which can provide additional insights. Detailed instructions can be found on the linked pages, but we've also included them below.

Note that when a new site is added to the system, all of its data will be automatically embargoed (only available to the uploader, and other users they specify) for 12 months. To modify this embargo period, or to modify user permissions for particular sites, send us an email.

Jump to Instructions for:
Uploading sensor data from core sites (those funded by the StreamPULSE grant)
Uploading sensor data from leveraged sites (all other sites)
Uploading manually collected data from any site
Revising or replacing previously uploaded data

After uploading, it's time to clean your data.


Uploading data for core sites

Core sites can upload raw datalogger files and/or pre-formatted data.


File naming

Name your upload file – REGIONID_SITEID_YYYY-MM-DD_LOGGERID.xxx – where

  • REGIONID is the name of your region (PR, FL, NC, WI, or AZ),
  • SITEID is your unique site name,
  • YYYY-MM-DD is the download date,
  • LOGGERID is the logger routing code,
    • CS: CR1000 data file
    • HD: Hobo DO logger
    • HW: Hobo water pressure logger
    • HA: Hobo air pressure logger
    • HP: Hobo light pendant logger
    • EM: Eureka Manta logger
    • XX: Calibrated and formatted data
  • and .xxx is the file extension.
    • Extension must be .dat if LOGGERID is CS. Otherwise must be .csv.
  • Note: LOGGERID may include numbers if you have several of the same logger type at one site. For example, LOGGERID could be CS4 or HA12.
  • Example filename: NC_Eno_2017-12-06_HP.csv

Calibrated and formatted data

You can upload raw data (from the datalogger) and/or calibrated data (e.g., turbidity in NTU, water level or discharge, etc.) at the same time.

If you modify a datalogger file to generate calibrated and derived variables, you must save it as a .csv with:

  • the _XX.csv extension (see LOGGERID above),
  • one header row followed directly by data rows, one row per timestamp,
  • the first column as a Date-Time stamp converted to UTC standard time and formatted as: YYYY-MM-DD HH:MM:SS, and
  • additional columns for each data variable.

Variables

Potential variables include:

  • Date-Time (UTC)
  • DO (mg/L)
  • Saturation DO (mg/L)
  • Water Temperature (°C)
  • Water Pressure (kPa)
  • Air Temperature (°C)
  • Air Pressure (kPa)
  • Depth (m)
  • Discharge (m3/s)
  • Velocity (m/s)
  • Light, PAR (μmol/m2/s)
  • Light, lux
  • Specific Conductivity (mS/cm or μS/cm)
  • pH
  • fDOM (frac)
  • fDOM (mV from sensor)
  • Turbidity (NTU)
  • Turbidity (mV from sensor)
  • Nitrate (mg/L)
  • CO2 (ppm)

Date formatting help for calibrated files

Date-time stamps can be challenging to format.

In R you can create a 'POSIXct' object. Below is an example of how to convert a date-time string to the correct format: 


datetimeorig <- "8/31/16 13:24:16" # can also be a vector

# In POSIX, 1. designate the format to match the original date time
# and 2. specify the timezone... a full list can be viewed by running OlsonNames()
dtval <- as.POSIXct(datetimeorig, format="%m/%d/%y %H:%M:%S", tz="EST")

# Then, just switch the display to UTC
attr(dtval,"tzone") <- "UTC"

# The output will be 2016-08-31T18:24:16Z

The as.POSIXct function can convert any date-time format and any time zone. For details on all of the format structure codes, see the R documentation.


In matlab you can create a date time string with the numeric values for your timestamp, accounting for the UTC offset:


time.UTC = -5; % UTC offset for EST
timeVec = [time.year time.month time.day time.hour-time.UTC time.min time.sec];
timeStr = datestr(timeVec,'yyyy-mm-dd HH:MM:SS'); % what you will save

In Excel you can modify the timestamp with a formula based on the timezone offset:


=TimeCell+(tzOffsetHours/24)

Then modify the cell format of the new column with a “custom” type to match YYYY-MM-DD HH:MM:SS (see documentation).

Be sure to put the modified date-time stamp as the first column in your exported .csv.


Saving files

Exporting a .csv from R is easy with the readr package, which saves files without row names and preserves the ISO date-time format:


library(readr)
write_csv(datatable, path="NC_Eno_2016-10-13_XX.csv")


Uploading data for leveraged sites

Leveraged sites need to format their data prior to upload as a .csv file with:

  • the specified file name (see below),
  • one header row followed directly by data rows, one row per timestamp,
  • the first column as a Date-Time stamp converted to UTC standard time and formatted as: YYYY-MM-DD HH:MM:SS, and
  • additional columns for each data variable.

If you are uploading updated data from an existing site, please only include the most recent data (since your last upload) in your file – our system saves the full uploaded file each time and redundant data just use extra bandwidth.


Variables

The minimum set of variables for upload are:

  • Date-Time (UTC)
  • Water Temperature (°C)
  • DO (mg/L)
  • Saturation DO (mg/L) and/or Air Pressure (kPa)
    • Air pressure can be estimated automatically for continental U.S.A.
  • Depth (m) and/or Discharge (m3/s)
    • Depth may also be referred to as "level" or "stage".

Bonus variables include:

  • Light, PAR (μmol/m2/s)
  • Light, lux
  • Specific Conductivity (mS/cm or μS/cm)
  • pH
  • fDOM (frac)
  • fDOM (mV from sensor)
  • Turbidity (NTU)
  • Turbidity (mV from sensor)
  • Nitrate (mg/L)
  • CO2 (ppm)
  • Water Pressure (kPa)
  • Air Temperature (°C)
  • Air Pressure (kPa)

Convert your variables to one of these formats prior to uploading (check your units!).


File naming

Name your upload file – REGIONID_SITEID_YYYY-MM-DD.csv – where

  • REGIONID is the name of your region (US: FIPS state code; International: ISO 3166-1 alpha-2 code),
  • SITEID is your chosen unique site name (only alphanumeric characters, any length you want), and
  • YYYY-MM-DD is the download date or the last date in the timeseries.

Date formatting help

Date-time stamps can be challenging to format.

In R you can create a 'POSIXct' object. Below is an example of how to convert a date-time string to the correct format: 


datetimeorig <- "8/31/16 13:24:16" # can also be a vector

# In POSIX, we 1. designate the format to match the original date time
# and 2. specify the timezone. A full list of timezones can be viewed with OlsonNames()
dtval <- as.POSIXct(datetimeorig, format="%m/%d/%y %H:%M:%S", tz="EST")

# Then, just switch the display to UTC
attr(dtval,"tzone") <- "UTC"

# The output will be 2016-08-31T18:24:16Z

The as.POSIXct function can convREGIONIDert any date-time format and any time zone. For details on all of the format structure codes, see the R documentation.


In matlab you can create a date time string with the numeric values for your timestamp, accounting for the UTC offset:


time.UTC = -5; % UTC offset for EST
timeVec = [time.year time.month time.day time.hour-time.UTC time.min time.sec];
timeStr = datestr(timeVec,'yyyy-mm-dd HH:MM:SS'); % what you will save

In Excel you can modify the timestamp with a formula based on the timezone offset:


=TimeCell+(tzOffsetHours/24)

Then modify the cell format of the new column with a “custom” type to match YYYY-MM-DD HH:MM:SS (see documentation).

Be sure to put the modified date-time stamp as the first column in your exported .csv.


Saving files

Exporting a .csv from R is easy with the readr package, which saves files without row names and preserves the ISO date-time format:

library(readr)
write_csv(datatable, path="NC_Eno_2016-10-13.csv")


Uploading Manually Collected Data


File naming


Name your file REGIONID_YYYY-MM-DD.csv, where

  • REGIONID is a two-letter abbreviation for your state, country, etc.
    • e.g. AZ for Arizona, PR for Puerto Rico, SE for Sweden
  • YYYY-MM-DD is a date that distinguishes this file from others.
    • This may be the date of the first or last observation in the file, the date the data were processed, or even an arbitrary date.

Example filename: NC_2017-12-06.csv


Data formatting

Data must be in standard CSV format, i.e.

  1. Records separated by newlines
  2. Fields separated by commas (not semicolons)
  3. Decimals demarcated by periods (not commas)
  4. Double quotes around fields with embedded commas, quotes, etc.

The file must contain:

  1. One header row followed directly by data rows
  2. Column 1:
    • Must be named "DateTime_UTC"
    • Must contain timestamps converted to UTC standard time
    • Must be formatted as: YYYY-MM-DD HH:MM:SS.
      See datetime formatting help below.
  3. Column 2:
    • Must be named "Sitecode"
    • Must contain unique abbreviations for the sites corresponding to your observations. These should match the abbreviations used to identify sensor sites (See the Sitelist tab.).
  4. One row per timestamp
  5. Additional columns for each data variable

Datetime formatting help

Date-time stamps can be challenging to format.

In R you can create a POSIXct object. Below is an example of converting a date-time string to the correct format: 

  datetimeorig <- "8/31/16 13:24:16" # can also be a vector

  # In POSIX, 1. designate the format to match the original date time
  #  and 2. specify the timezone... a full list can be viewed by running OlsonNames()
  dtval <- as.POSIXct(datetimeorig, format="%m/%d/%y %H:%M:%S", tz="EST")

  # Then, just switch the display to UTC
  attr(dtval,"tzone") <- "UTC"

  # The output will be 2016-08-31T18:24:16Z

The as.POSIXct function can convert any date-time format and any time zone. For details on all of the format structure codes, see the R documentation.

In matlab you can create a date time string with the numeric values for your timestamp, accounting for the UTC offset: 

  time.UTC = -5; % UTC offset for EST
  timeVec = [time.year time.month time.day time.hour-time.UTC time.min time.sec];
  timeStr = datestr(timeVec,'yyyy-mm-dd HH:MM:SS'); % what you will save

In Excel you can modify the timestamp with a formula based on the timezone offset: 

=TimeCell+(tzOffsetHours/24)

Then modify the cell format of the new column with a “custom” type to match YYYY-MM-DD HH:MM:SS ( see documentation).

Be sure to put the modified date-time stamp as the first column in your exported .csv file.


Saving files

Exporting a .csv file from R is easy with the readr package, which saves files without row names and preserves the ISO date-time format:

  library(readr)
  write_csv(datatable, path="NC_Eno_2016-10-13_XX.csv")


Revising Data

To revise previously uploaded data, you must upload a revised file with the same name as the original. All data previously associated with that file will be replaced in our database. The original file will still be stored on our server, however.

Here's how it works:

  1. Make a local copy of the CSV or DAT file that's being revised.
  2. Modify the name of the copy that will not receive data edits. You may want to call it something like AA_BB_2017-01-01_XX_unrevised.csv.
  3. Change erroneous data in the other copy. Make sure its name is identical to the originally uploaded file.
  4. Upload this revised copy. Make sure to check the box below.

NOTICE: Currently there can be unintended consequences if you upload the same variable, from the same place and time, in multiple files (i.e. files with different names). We realize it can be inconvenient to locate original files in order to revise StreamPULSE data, but at present that is the only way. We will enable more flexible data revision in a future update.

If you have any questions, please send us an email.