Earlier this year I have posted an article on how I managed to gather and persist calories from FDDB.info. As I have grown more interested in macro-nutrients, I had a look into extending my application to gather more data. Unfortunately this resulted in a complete rewrite, as the approach I took in March relied on the CSV export from FDDB, which does not contain data about consumed sugars.
Looking into FDDB and their exports I soon realized I had parse the HTML of their website, because the needed data is only present on there and not contained in any structured exports. Making use of my knowledge about their site from my earlier exporting approach I started up IntelliJ and decided to create a second Micronaut application, after my first steps with it in the Datadog metrics project. The tool I have created can be found on my Github account:
This tool exports the following data on a daily basis: kcal, fat, carbs, sugar, protein, fiber. To achieve this Micronaut runs a scheduler which gathers the data for the day before at night. The retrieved data is then written to a Postgres database. To gather data of previous days, weeks and months I implemented a HTTP endpoint which can be called by appending /batch
to the context route and sending a POST request containing the following JSON:
{ "fromDate": "2021-05-13", "toDate": "2021-08-18" }
The data is then persisted to the Postgres database for each day. Having all the data in Postgres present makes it quite easy to create some graphs to track the consumption of macro-nutrients. Grafana makes this quite easy:


For the prerequisites see my earlier post explaining how to obtain the necessary cookies for FDDB.info to allow HTTP requests to access your login.
braucht das dockerimage die gleichen ENV variablen wie die alte version?
leider connected postgres bei mir nicht…
logs
https://file.io/Dt80xdbIZZVJ
thx!
Hey, nein, das sind andere Variablen, sofern du mit alte Version das Python Skript meinst. Ich habe hier die Readme aktualisiert mit einem Beispiel der Umgebungsvariablen: https://github.com/itobey/fddb-exporter#using-the-image
danke bin etwas weitergekommen 🙂
ich seh bei fddb.info meine entraege aber der log schreibt nur
15:03:25.074 [default-nioEventLoopGroup-1-3] ERROR com.itobey.adapter.api.fddb.exporter.service.HtmlParser – no value found for kcal, there might be no data available for the specified day,
15:03:25.074 [default-nioEventLoopGroup-1-3] WARN com.itobey.adapter.api.fddb.exporter.service.ManualExportService – data for date 2023-03-17 cannot be parsed, skipping this day
Cool 🙂 Ich vermute, dass dein Cookie oder Basicauth nicht passt. Ich werde in den nächsten Tagen / Wochen mal eine überarbeitete Version veröffentlichen, um da eine bessere Fehlermeldung zu loggen.
Ich habe mir das Thema heute mal angeschaut und das Problem ist tatsächlich eine Änderung von FDDB. Hier hat sich das Placement des Login Buttons geändert und deshalb ist mein Fehlerhandling nicht angeschlagen. Ich habe einen Fix veröffentlicht, du kannst dir das Image unter der Version 0.2.1 laden, dann bekommst du eine korrekte Meldung, wenn der Login nicht funktioniert.