GuentherWuchterl: Die Seite wurde neu angelegt: „= Database for the Lightmeter Network = The lightmeter data is centrally collected at a Zentrum für Astronomie Heidelberg in the context of the German Astrophysical Virtual Observatory (GAVO). '''The Lightmeter database:''' [http://vo.uni-hd.de/lightweather ''Lightweather'' at GAVO] '''Stations in the datase:''' [http://dc.zah.uni-heidelberg.de/system/adql/query/form?__nevow_form__=genForm&query=select%20*%20from%20lightmeter.stations&timeo…“

2023-08-27T13:55:47Z

Die Seite wurde neu angelegt: „= Database for the Lightmeter Network = The lightmeter data is centrally collected at a Zentrum für Astronomie Heidelberg in the context of the German Astrophysical Virtual Observatory (GAVO). '''The Lightmeter database:''' [http://vo.uni-hd.de/lightweather ''Lightweather'' at GAVO] '''Stations in the datase:''' [http://dc.zah.uni-heidelberg.de/__system__/adql/query/form?__nevow_form__=genForm&query=select%20*%20from%20lightmeter.stations&timeo…“

Neue Seite

= Database for the Lightmeter Network =

The lightmeter data is centrally collected at a Zentrum für Astronomie Heidelberg in the context of the German Astrophysical Virtual Observatory (GAVO).

'''The Lightmeter database:''' [http://vo.uni-hd.de/lightweather ''Lightweather'' at GAVO]
'''Stations in the datase:''' [http://dc.zah.uni-heidelberg.de/__system__/adql/query/form?__nevow_form__=genForm&query=select%20*%20from%20lightmeter.stations&timeout=5&_FORMAT=HTML&submit=Go List with metadata as coordinates and calibrations]
'''Recent data:''' [http://dc.zah.uni-heidelberg.de/lightmeter/q/weather Query form and graph of recent uploads]

Data are submitted by HTTP uploads to that location. To make this work, some rules must be followed.

== Station Metadata ==

To make your data useful, it must be accompanied by some metadata. If you
use skysensor (in a version recent enough), you can submit its ini-file that is located in the same directory as the executable. Otherwise, fill out the following template:

<pre>
fullname: (Short free text, not more than, say, 70 characters)
stationId: (The station id)
long: (longitude in decimal degrees, east is positive)
lat: (latitude in decimal degrees)
height: (in meters above sea level)
</pre>

Mail your file to gavo@ari.uni-heidelberg.de. We will reply by providing you an
access key that you will need to upload the data.

The station id consists of the ISO country code, a designation of the location
and a number; e.g., DE_HEIDELBERG_1; only letters, digits and underscores are
allowed. On metadata submission, choose your own identifier. We may have to
change your station id to avoid name clashes, though.

'''Important''': If you move your sensor, it ''must'' receive a new station id, and
you ''must'' submit a new metadata set. Analoguosly, if you change your sensor,
you ''must'' obtain a new station id. A "station" is the combination of a
specific location and a specific sensor.

== Data Upload ==

After you have received your access key (and possibly a revised station id),
you can begin uploading. This is done by HTTP uploads. You should try
uploading manually first. To do this, go to
http://vo.uni-hd.de/lightmeter/q/upload/custom/STATIONID/ACCESSKEY
(where you must replace STATIONID and ACCESSKEY with what was sent to you,
of course).

=== Upload Formats ===

You can upload both CSV and "text" format. Text format consists of lines

<pre>
date time(utc) temperature count status
</pre>

Such a file could look like this:

<pre>
2009-05-29 00:00:01 11,3 235160 1
2009-05-29 00:00:02 11,3 235240 1
2009-05-29 00:00:03 11,3 235320 1
2009-05-29 00:00:04 11,3 235320 1
2009-05-29 00:00:05 11,3 235264 1
2009-05-29 00:00:06 11,3 234864 1
2009-05-29 00:00:07 11,3 235360 1
</pre>

CSV format has semicolons as separators and must not have headers. It should
look like this:

<pre>
27.04.2009;12:31:14;36,9;°C;1942560;ok;
27.04.2009;12:31:15;37,0;°C;1947960;ok;
27.04.2009;12:31:16;37,0;°C;1951800;ok;
27.04.2009;12:31:17;36,9;°C;1943880;ok;
27.04.2009;12:31:18;37,0;°C;1947960;ok;
27.04.2009;12:31:19;36,9;°C;1956960;ok;
</pre>

We will accept floating point numbers with both commas and decimal points as decimal separators.

You can gzip your submissions before transfer.

The receiving software is somewhat naive and infers the content from the
file name extension. Legal extensions are:

* .txt -- text format
* .txt.gz -- gzip compressed text format
* .csv -- csv format
* .csv.gz -- gzip compressed csv format

Make sure you follow these conventions.

You can also upload ZIP archives of such files. The only legal extension here is zip (lower case!).

=== Automatic Uploads ===

Of course, manual uploads will become quite tedious. Therefore, we provide an
automatic upload facility. While you can use anything that can do HTTP uploads
(your the file goes into the inFile key, and you must give a __nevow_form__ key
with the value upload), [http://vo.uni-hd.de/lightmeter/q/upload/static/src/uploadLM.py this python script] does what you want (including
automatic transfer compression). The program is written for python 2.x; unless you have a ''very'' good reason, you should not use python 3.x just yet, and if you have one, please feed back the (minor) patches to make the uploader work with 3.x (while keeping it ok for 2.x).

To use it, put a file named "stationinfo" into the directory from which you
will upload. It must contain the station id, a blank, and the access key.

Then, in a shell, say (adapt for the location of your python interpreter and
the script as necessary)

<pre>python uploadLM.py FILE1 FILE2...</pre>

We expect uploading will be included in some of the readout software.

== Calibration ==

Your data will not immediately appear in the database since we first have to
determine the calibration of your sensor. To let us do that, please wait for
clear skies in the morning or evening (while taking data, of course). After you have uploaded the data
obtained during such a clear dusk or dawn, notfiy TBD who will then derive
a calibration from the uploaded data. Once the calibration is done,
your data will show up in the database.

A step by step description of the calibration process is described at
[[Calibration example Linz]].

==Topics for possible preprocessing on upload==

# time-zones --> UTC with tzdata
# homogenize decimal separator to "." (issue for Temperature)
# correct temperature-readings T-512 for T>256.
--[[Benutzer:GuentherWuchterl|GuentherWuchterl]] 11:02, 24. Feb. 2010 (UTC)

# homogenize date format (Y-m-d vs. d.m.Y)
# Where? Who? [from my view: preferably at the user's; I wouldn't want to keep corrected and uncorrected files on my side]

-- Markus 2010-02-24

# Are there top 3 (T-512) has a solution? The problem is known from the manufacturer.
-- Hagen

==Test queries==

[http://dc.zah.uni-heidelberg.de/__system__/adql/query/form?__nevow_form__=genForm&query=select%20*%20from%20lightmeter.stations&timeout=5&_FORMAT=HTML&submit=Go Stations in DB]

[http://dc.zah.uni-heidelberg.de/lightmeter/q/weather/form?__nevow_form__=genForm&date__day=10&date__month=10&date__year=2009&time=05%3A00%3A00&within=1000&_DBOPTIONS_ORDER=&_DBOPTIONS_LIMIT=100&_FORMAT=HTML&submit=Go Test query at GVO - to plot data scroll down and select VOplot as format]

[http://dc.zah.uni-heidelberg.de/lightmeter/q/upload/static/ Station uploads]

==Examples for how to use GAVO from other fields==

http://wwwas.oats.inaf.it/aidawp5/index_eng.html?fsize=medium

e.g.: Asteroids in the Solar System
http://wwwas.oats.inaf.it/aidawp5/download/eng_es9_asteroids.pdf

GAVO database - Versionsgeschichte