Hello everyone,
An important question has arisen regarding the evaluation methodology for our models in the phase recognition task. We must determine whether the model evaluation will be conducted online or offline. This means the model can use future and past frames for inference (offline) or is limited to only current and past information at each timestep (online).
Dear Santiago,
Thank you very much for your request. For the evaluation of the phase recognition task, only the frames from the past and the current frame can be used, i.e., it is an online setting, and you do not have access to future frames.
Best regards,
Tobias Rueckert
@tobias-ruckert I haven't seen any evaluation codes for phase recognition in the evaluation.py, and what's the expected output format for phase recognition? Are you still working on releasing them? Thanks!
Dear Yiping,
Thank you very much for your question. The file "phakir_evaluation.py" is intended to evaluate the input test data with your model and to save the results in the "outputs" folder. The calculation of the metrics is then done afterward using these outputs, not directly in the "phakir_evaluation.py" file. The metrics used for each task are described on our website, see https://phakir.re-mic.de/tasks/ .
The expected output format for each task is exactly the same as the structure of the ground truth data, in the case of phase recognition this would be a CSV file, which has the exact same structure as the CSV files of the training data. These specifications have already been published and can be found in the README of the PhaKIR submission template repository, see https://github.com/schnadoslin/PhaKIR_Submission_Template/tree/main . When submitting, please make sure that the folder structure, file naming, etc. matches the specifications in the README of the repository.
Best regards,
Tobias Rueckert
@tobias-ruckert Thanks for the reply! I understand now. Just to double check, so the outputs of phase recognition should also be in 25 fps? (as for the other tasks it's 1fps?)
Dear Yiping,
Yes, you are correct, the outputs for the phase recognition task should be provided at 25 fps, and for the instance segmentation and keypoint estimation tasks, the outputs should be provided at 1 fps.
Best regards,
Tobias Rueckert