Ggtcf

How can I raise concerns with a new DM about XP splitting?

Identify a stage play about a VR experience in which participants are encouraged to simulate performing horrific activities

node command while defining a coordinate in TikZ

Should my PhD thesis be submitted under my legal name?

Perfect riffle shuffles

A social experiment. What is the worst that can happen?

Can the harmonic series explain the origin of the major scale?

How did Monica know how to operate Carol's "designer"?

Superhero words!

My boss asked me to take a one-day class, then signs it up as a day off

Pronouncing Homer as in modern Greek

Invariance of results when scaling explanatory variables in logistic regression, is there a proof?

Visiting the UK as unmarried couple

For airliners, what prevents wing strikes on landing in bad weather?

What to do when my ideas aren't chosen, when I strongly disagree with the chosen solution?

Is exact Kanji stroke length important?

I2C signal and power over long range (10meter cable)

Can a malicious addon access internet history and such in chrome/firefox?

Can I Retrieve Email Addresses from BCC?

How will losing mobility of one hand affect my career as a programmer?

Teaching indefinite integrals that require special-casing

The most efficient algorithm to find all possible integer pairs which sum to a given integer

In Star Trek IV, why did the Bounty go back to a time when whales were already rare?

What (else) happened July 1st 1858 in London?

what is causing F1-score high but Accuracy low in a deep learning model?

What is the relationship between loss and validation accuracy?Is it always possible for validation accuracy to be as high as training accuracy?Reporting accuracy and loss issues with MonitoredTrainingSessionthe training accuracy steadily increase, but training loss decrease and then increasePython Keras LSTM learning converges too fast on high lossTest Accuracy Increases Whilst Loss IncreasesWhen to keep learned weights after change deep learning model/hyperparametersError in model performance metricsCNN with Tensorflow, low accuracy on CIFAR-10 and not improvingAccuracy in a CNN model never goes high for training and validation set

0

i'm using BERT base-uncased model to train NER on conll-2003 dataset. i just used BertForTokenClassification (from huggingface) for training which is kind of considering final sequence-layer and then adding final linear layer. where i'm able to produce below results.

with 6 epoch with train/dev data size:: 6973/1739
Test F1-Score: 0.8455102584598987
'test_loss': 0.18759359930737468, 'test_accuracy': 0.42335164835164835, 'global_step': 1308, 'loss': 0.03054473980611891
Validation F1-Score: 0.8771035676507356
'eval_loss': 0.13038920708013477, 'eval_accuracy': 0.4910168195718655, 'global_step': 1308, 'loss': 0.03054473980611891

for finding loss i'm using below functions.

def flat_accuracy(preds, labels):
 pred_flat = np.argmax(preds, axis=2).flatten()
 labels_flat = labels.flatten()
 return np.sum(pred_flat == labels_flat) / len(labels_flat)
for each_batch:
 tmp_eval_accuracy = flat_accc(pred_xx, label_ids_xx)
 eval_accuracy += tmp_eval_accuracy
 nb_eval_steps += 1
eval_accuracy = eval_accuracy / nb_eval_steps

if you have seen above results, it's really bad in terms of accuracy. my question is the method i'm using for finding accuracy is it right or wrong ? i believe it's right because it's just matching number of labels matched out of total labels. and finally sum of each small batch accuracy divide with total batch count.

but if you see, F1-score is coming high. and for F1 score i used (from seqeval.metrics import f1_score)

please tell me what are the possible causes/meaning behind it ?
and how can i know whether my model learned properly or not ? like it should have faced any bias-variance trade-off.

please let me know if you want more information for clarity..
Thanks in advance.

deep-learning ner

share|improve this question

asked Mar 8 at 8:06

DONDON

206

add a comment | 

0

i'm using BERT base-uncased model to train NER on conll-2003 dataset. i just used BertForTokenClassification (from huggingface) for training which is kind of considering final sequence-layer and then adding final linear layer. where i'm able to produce below results.

with 6 epoch with train/dev data size:: 6973/1739
Test F1-Score: 0.8455102584598987
'test_loss': 0.18759359930737468, 'test_accuracy': 0.42335164835164835, 'global_step': 1308, 'loss': 0.03054473980611891
Validation F1-Score: 0.8771035676507356
'eval_loss': 0.13038920708013477, 'eval_accuracy': 0.4910168195718655, 'global_step': 1308, 'loss': 0.03054473980611891

for finding loss i'm using below functions.

def flat_accuracy(preds, labels):
 pred_flat = np.argmax(preds, axis=2).flatten()
 labels_flat = labels.flatten()
 return np.sum(pred_flat == labels_flat) / len(labels_flat)
for each_batch:
 tmp_eval_accuracy = flat_accc(pred_xx, label_ids_xx)
 eval_accuracy += tmp_eval_accuracy
 nb_eval_steps += 1
eval_accuracy = eval_accuracy / nb_eval_steps

if you have seen above results, it's really bad in terms of accuracy. my question is the method i'm using for finding accuracy is it right or wrong ? i believe it's right because it's just matching number of labels matched out of total labels. and finally sum of each small batch accuracy divide with total batch count.

but if you see, F1-score is coming high. and for F1 score i used (from seqeval.metrics import f1_score)

please tell me what are the possible causes/meaning behind it ?
and how can i know whether my model learned properly or not ? like it should have faced any bias-variance trade-off.

please let me know if you want more information for clarity..
Thanks in advance.

deep-learning ner

share|improve this question

asked Mar 8 at 8:06

DONDON

206

add a comment | 

i'm using BERT base-uncased model to train NER on conll-2003 dataset. i just used BertForTokenClassification (from huggingface) for training which is kind of considering final sequence-layer and then adding final linear layer. where i'm able to produce below results.

with 6 epoch with train/dev data size:: 6973/1739
Test F1-Score: 0.8455102584598987
'test_loss': 0.18759359930737468, 'test_accuracy': 0.42335164835164835, 'global_step': 1308, 'loss': 0.03054473980611891
Validation F1-Score: 0.8771035676507356
'eval_loss': 0.13038920708013477, 'eval_accuracy': 0.4910168195718655, 'global_step': 1308, 'loss': 0.03054473980611891

for finding loss i'm using below functions.

def flat_accuracy(preds, labels):
 pred_flat = np.argmax(preds, axis=2).flatten()
 labels_flat = labels.flatten()
 return np.sum(pred_flat == labels_flat) / len(labels_flat)
for each_batch:
 tmp_eval_accuracy = flat_accc(pred_xx, label_ids_xx)
 eval_accuracy += tmp_eval_accuracy
 nb_eval_steps += 1
eval_accuracy = eval_accuracy / nb_eval_steps

if you have seen above results, it's really bad in terms of accuracy. my question is the method i'm using for finding accuracy is it right or wrong ? i believe it's right because it's just matching number of labels matched out of total labels. and finally sum of each small batch accuracy divide with total batch count.

but if you see, F1-score is coming high. and for F1 score i used (from seqeval.metrics import f1_score)

please tell me what are the possible causes/meaning behind it ?
and how can i know whether my model learned properly or not ? like it should have faced any bias-variance trade-off.

please let me know if you want more information for clarity..
Thanks in advance.

deep-learning ner

share|improve this question

asked Mar 8 at 8:06

DONDON

206

with 6 epoch with train/dev data size:: 6973/1739
Test F1-Score: 0.8455102584598987
'test_loss': 0.18759359930737468, 'test_accuracy': 0.42335164835164835, 'global_step': 1308, 'loss': 0.03054473980611891
Validation F1-Score: 0.8771035676507356
'eval_loss': 0.13038920708013477, 'eval_accuracy': 0.4910168195718655, 'global_step': 1308, 'loss': 0.03054473980611891

def flat_accuracy(preds, labels):
 pred_flat = np.argmax(preds, axis=2).flatten()
 labels_flat = labels.flatten()
 return np.sum(pred_flat == labels_flat) / len(labels_flat)
for each_batch:
 tmp_eval_accuracy = flat_accc(pred_xx, label_ids_xx)
 eval_accuracy += tmp_eval_accuracy
 nb_eval_steps += 1
eval_accuracy = eval_accuracy / nb_eval_steps

share|improve this question

asked Mar 8 at 8:06

DONDON

206

share|improve this question

asked Mar 8 at 8:06

DONDON

206

asked Mar 8 at 8:06

DONDON

206

asked Mar 8 at 8:06

DONDON

206