MssqlOperater not executing the sql statements + apache airflow + dag2019 Community Moderator ElectionAirflow does not update the progress of a dag/task to be completed even though the dag/task has actually completedAirflow scheduler not picking up dagsairflow - all tasks being queued and not moved to executionHow to validate airflow DAG with customer operator?How to force jinja templating on airflow variable?DAGs not clickable on Google Cloud Composer webserver, but working fine on a local AirflowApache - Airflow 1.10.1 don't start a jobAirflow on_success_callback() does not runHow to indicate config for Airflow when using run_as_user?Multiple BashOperator in Airflow doesn't recognize the current folder
Is it a Cyclops number? "Nobody" knows!
The meaning of ‘otherwise’
Should I take out a loan for a friend to invest on my behalf?
Confusion about Complex Continued Fraction
Do I really need to have a scientific explanation for my premise?
After `ssh` without `-X` to a machine, is it possible to change `$DISPLAY` to make it work like `ssh -X`?
Can one live in the U.S. and not use a credit card?
Shifting between bemols and diesis in the key signature
What is better: yes / no radio, or simple checkbox?
What is the generally accepted pronunciation of “topoi”?
Does Christianity allow for believing on someone else's behalf?
What do *foreign films* mean for an American?
What materials can be used to make a humanoid skin warm?
Why does cron require MTA for logging?
What are some noteworthy "mic-drop" moments in math?
Does an unused member variable take up memory?
Was it really inappropriate to write a pull request for the company I interviewed with?
PTIJ: Why does only a Shor Tam ask at the Seder, and not a Shor Mu'ad?
Is it possible to avoid unpacking when merging Association?
Gaining more land
What is the population of Romulus in the TNG era?
School performs periodic password audits. Is my password compromised?
Are small insurances worth it?
Possible to detect presence of nuclear bomb?
MssqlOperater not executing the sql statements + apache airflow + dag
2019 Community Moderator ElectionAirflow does not update the progress of a dag/task to be completed even though the dag/task has actually completedAirflow scheduler not picking up dagsairflow - all tasks being queued and not moved to executionHow to validate airflow DAG with customer operator?How to force jinja templating on airflow variable?DAGs not clickable on Google Cloud Composer webserver, but working fine on a local AirflowApache - Airflow 1.10.1 don't start a jobAirflow on_success_callback() does not runHow to indicate config for Airflow when using run_as_user?Multiple BashOperator in Airflow doesn't recognize the current folder
below is the dag I have written, when I ran the dag manually/scheduled it the tasks show success but its not executing sql statements at backend . I have copied both dag code and log associated with the task . can anyone help me on this ?
from airflow import DAG
from airflow.operators.mssql_operator
import MsSqlOperator from datetime
import datetime, timedelta
from airflow.operators.bash_operator import BashOperator
default_args =
'owner': 'santosh',
'depends_on_past': False,
'email': ['santosh.dadisetti@gmail.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'start_date': datetime(2019,3,4)
dag = DAG('MSSQL_server_test1', default_args=default_args)
task1 = MsSqlOperator(
task_id='MSSQL_Task',
mssql_conn_id='mssql_hook',
database='sampledb',
sql='INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[Table1]',
dag=dag)
task2 = BashOperator(
task_id='taskid_2',
bash_command='echo "Hello World from Task 2"',
dag=dag)
task1 >> task2
Log
*** Reading local file: /root/airflow/logs/MSSQL_server_test1/MSSQL_Task/2019-03-06T21:09:07.392474+00:00/1.log
[2019-03-06 21:09:24,674] models.py:1359 INFO - Dependencies all met for TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [queued]
[2019-03-06 21:09:24,833] models.py:1359 INFO - Dependencies all met for TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [queued]
[2019-03-06 21:09:24,833] models.py:1571 INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 2
--------------------------------------------------------------------------------
[2019-03-06 21:09:25,414] models.py:1593 INFO - Executing <Task(MsSqlOperator): MSSQL_Task> on 2019-03-06T21:09:07.392474+00:00
[2019-03-06 21:09:25,415] base_task_runner.py:118 INFO - Running: ['bash', '-c', 'airflow run MSSQL_server_test1 MSSQL_Task 2019-03-06T21:09:07.392474+00:00 --job_id 47 --raw -sd DAGS_FOLDER/mssql_1.py --cfg_path /tmp/tmppofyzjrw']
[2019-03-06 21:09:26,802] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:26,801] settings.py:174 INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=29200
[2019-03-06 21:09:27,615] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:27,613] __init__.py:51 INFO - Using executor SequentialExecutor
[2019-03-06 21:09:29,164] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:29,162] models.py:273 INFO - Filling up the DagBag from /root/airflow/dags/mssql.py
[2019-03-06 21:09:30,664] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:30,663] cli.py:520 INFO - Running <TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [running]> on host XXXXXX
[2019-03-06 21:09:32,126] mssql_operator.py:54 INFO - Executing: INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[table1]
[2019-03-06 21:09:32,608] logging_mixin.py:95 INFO - [2019-03-06 21:09:32,606] base_hook.py:83 INFO - Using connection to: id: mssql_hook. Host: <host>, Port: 1433, Schema: None, Login: usr_airflow, Password: XXXXXXXX, extra:
[2019-03-06 21:09:32,985] logging_mixin.py:95 INFO - [2019-03-06 21:09:32,985] dbapi_hook.py:166 INFO - INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[Table1]
[2019-03-06 21:09:34,725] logging_mixin.py:95 INFO - [2019-03-06 21:09:34,722] jobs.py:2527 INFO - Task exited with return code 0
airflow airflow-scheduler
add a comment |
below is the dag I have written, when I ran the dag manually/scheduled it the tasks show success but its not executing sql statements at backend . I have copied both dag code and log associated with the task . can anyone help me on this ?
from airflow import DAG
from airflow.operators.mssql_operator
import MsSqlOperator from datetime
import datetime, timedelta
from airflow.operators.bash_operator import BashOperator
default_args =
'owner': 'santosh',
'depends_on_past': False,
'email': ['santosh.dadisetti@gmail.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'start_date': datetime(2019,3,4)
dag = DAG('MSSQL_server_test1', default_args=default_args)
task1 = MsSqlOperator(
task_id='MSSQL_Task',
mssql_conn_id='mssql_hook',
database='sampledb',
sql='INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[Table1]',
dag=dag)
task2 = BashOperator(
task_id='taskid_2',
bash_command='echo "Hello World from Task 2"',
dag=dag)
task1 >> task2
Log
*** Reading local file: /root/airflow/logs/MSSQL_server_test1/MSSQL_Task/2019-03-06T21:09:07.392474+00:00/1.log
[2019-03-06 21:09:24,674] models.py:1359 INFO - Dependencies all met for TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [queued]
[2019-03-06 21:09:24,833] models.py:1359 INFO - Dependencies all met for TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [queued]
[2019-03-06 21:09:24,833] models.py:1571 INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 2
--------------------------------------------------------------------------------
[2019-03-06 21:09:25,414] models.py:1593 INFO - Executing <Task(MsSqlOperator): MSSQL_Task> on 2019-03-06T21:09:07.392474+00:00
[2019-03-06 21:09:25,415] base_task_runner.py:118 INFO - Running: ['bash', '-c', 'airflow run MSSQL_server_test1 MSSQL_Task 2019-03-06T21:09:07.392474+00:00 --job_id 47 --raw -sd DAGS_FOLDER/mssql_1.py --cfg_path /tmp/tmppofyzjrw']
[2019-03-06 21:09:26,802] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:26,801] settings.py:174 INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=29200
[2019-03-06 21:09:27,615] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:27,613] __init__.py:51 INFO - Using executor SequentialExecutor
[2019-03-06 21:09:29,164] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:29,162] models.py:273 INFO - Filling up the DagBag from /root/airflow/dags/mssql.py
[2019-03-06 21:09:30,664] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:30,663] cli.py:520 INFO - Running <TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [running]> on host XXXXXX
[2019-03-06 21:09:32,126] mssql_operator.py:54 INFO - Executing: INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[table1]
[2019-03-06 21:09:32,608] logging_mixin.py:95 INFO - [2019-03-06 21:09:32,606] base_hook.py:83 INFO - Using connection to: id: mssql_hook. Host: <host>, Port: 1433, Schema: None, Login: usr_airflow, Password: XXXXXXXX, extra:
[2019-03-06 21:09:32,985] logging_mixin.py:95 INFO - [2019-03-06 21:09:32,985] dbapi_hook.py:166 INFO - INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[Table1]
[2019-03-06 21:09:34,725] logging_mixin.py:95 INFO - [2019-03-06 21:09:34,722] jobs.py:2527 INFO - Task exited with return code 0
airflow airflow-scheduler
Are you 100% sure that your query behaves correctly ,especially MSSQL's []-SquareBrackets syntax? And did you try it with setting the operators argumentautocommit=True
and the statements actually get's committed?
– dorvak
Mar 7 at 8:48
add a comment |
below is the dag I have written, when I ran the dag manually/scheduled it the tasks show success but its not executing sql statements at backend . I have copied both dag code and log associated with the task . can anyone help me on this ?
from airflow import DAG
from airflow.operators.mssql_operator
import MsSqlOperator from datetime
import datetime, timedelta
from airflow.operators.bash_operator import BashOperator
default_args =
'owner': 'santosh',
'depends_on_past': False,
'email': ['santosh.dadisetti@gmail.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'start_date': datetime(2019,3,4)
dag = DAG('MSSQL_server_test1', default_args=default_args)
task1 = MsSqlOperator(
task_id='MSSQL_Task',
mssql_conn_id='mssql_hook',
database='sampledb',
sql='INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[Table1]',
dag=dag)
task2 = BashOperator(
task_id='taskid_2',
bash_command='echo "Hello World from Task 2"',
dag=dag)
task1 >> task2
Log
*** Reading local file: /root/airflow/logs/MSSQL_server_test1/MSSQL_Task/2019-03-06T21:09:07.392474+00:00/1.log
[2019-03-06 21:09:24,674] models.py:1359 INFO - Dependencies all met for TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [queued]
[2019-03-06 21:09:24,833] models.py:1359 INFO - Dependencies all met for TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [queued]
[2019-03-06 21:09:24,833] models.py:1571 INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 2
--------------------------------------------------------------------------------
[2019-03-06 21:09:25,414] models.py:1593 INFO - Executing <Task(MsSqlOperator): MSSQL_Task> on 2019-03-06T21:09:07.392474+00:00
[2019-03-06 21:09:25,415] base_task_runner.py:118 INFO - Running: ['bash', '-c', 'airflow run MSSQL_server_test1 MSSQL_Task 2019-03-06T21:09:07.392474+00:00 --job_id 47 --raw -sd DAGS_FOLDER/mssql_1.py --cfg_path /tmp/tmppofyzjrw']
[2019-03-06 21:09:26,802] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:26,801] settings.py:174 INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=29200
[2019-03-06 21:09:27,615] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:27,613] __init__.py:51 INFO - Using executor SequentialExecutor
[2019-03-06 21:09:29,164] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:29,162] models.py:273 INFO - Filling up the DagBag from /root/airflow/dags/mssql.py
[2019-03-06 21:09:30,664] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:30,663] cli.py:520 INFO - Running <TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [running]> on host XXXXXX
[2019-03-06 21:09:32,126] mssql_operator.py:54 INFO - Executing: INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[table1]
[2019-03-06 21:09:32,608] logging_mixin.py:95 INFO - [2019-03-06 21:09:32,606] base_hook.py:83 INFO - Using connection to: id: mssql_hook. Host: <host>, Port: 1433, Schema: None, Login: usr_airflow, Password: XXXXXXXX, extra:
[2019-03-06 21:09:32,985] logging_mixin.py:95 INFO - [2019-03-06 21:09:32,985] dbapi_hook.py:166 INFO - INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[Table1]
[2019-03-06 21:09:34,725] logging_mixin.py:95 INFO - [2019-03-06 21:09:34,722] jobs.py:2527 INFO - Task exited with return code 0
airflow airflow-scheduler
below is the dag I have written, when I ran the dag manually/scheduled it the tasks show success but its not executing sql statements at backend . I have copied both dag code and log associated with the task . can anyone help me on this ?
from airflow import DAG
from airflow.operators.mssql_operator
import MsSqlOperator from datetime
import datetime, timedelta
from airflow.operators.bash_operator import BashOperator
default_args =
'owner': 'santosh',
'depends_on_past': False,
'email': ['santosh.dadisetti@gmail.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'start_date': datetime(2019,3,4)
dag = DAG('MSSQL_server_test1', default_args=default_args)
task1 = MsSqlOperator(
task_id='MSSQL_Task',
mssql_conn_id='mssql_hook',
database='sampledb',
sql='INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[Table1]',
dag=dag)
task2 = BashOperator(
task_id='taskid_2',
bash_command='echo "Hello World from Task 2"',
dag=dag)
task1 >> task2
Log
*** Reading local file: /root/airflow/logs/MSSQL_server_test1/MSSQL_Task/2019-03-06T21:09:07.392474+00:00/1.log
[2019-03-06 21:09:24,674] models.py:1359 INFO - Dependencies all met for TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [queued]
[2019-03-06 21:09:24,833] models.py:1359 INFO - Dependencies all met for TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [queued]
[2019-03-06 21:09:24,833] models.py:1571 INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 2
--------------------------------------------------------------------------------
[2019-03-06 21:09:25,414] models.py:1593 INFO - Executing <Task(MsSqlOperator): MSSQL_Task> on 2019-03-06T21:09:07.392474+00:00
[2019-03-06 21:09:25,415] base_task_runner.py:118 INFO - Running: ['bash', '-c', 'airflow run MSSQL_server_test1 MSSQL_Task 2019-03-06T21:09:07.392474+00:00 --job_id 47 --raw -sd DAGS_FOLDER/mssql_1.py --cfg_path /tmp/tmppofyzjrw']
[2019-03-06 21:09:26,802] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:26,801] settings.py:174 INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=29200
[2019-03-06 21:09:27,615] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:27,613] __init__.py:51 INFO - Using executor SequentialExecutor
[2019-03-06 21:09:29,164] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:29,162] models.py:273 INFO - Filling up the DagBag from /root/airflow/dags/mssql.py
[2019-03-06 21:09:30,664] base_task_runner.py:101 INFO - Job 47: Subtask MSSQL_Task [2019-03-06 21:09:30,663] cli.py:520 INFO - Running <TaskInstance: MSSQL_server_test1.MSSQL_Task 2019-03-06T21:09:07.392474+00:00 [running]> on host XXXXXX
[2019-03-06 21:09:32,126] mssql_operator.py:54 INFO - Executing: INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[table1]
[2019-03-06 21:09:32,608] logging_mixin.py:95 INFO - [2019-03-06 21:09:32,606] base_hook.py:83 INFO - Using connection to: id: mssql_hook. Host: <host>, Port: 1433, Schema: None, Login: usr_airflow, Password: XXXXXXXX, extra:
[2019-03-06 21:09:32,985] logging_mixin.py:95 INFO - [2019-03-06 21:09:32,985] dbapi_hook.py:166 INFO - INSERT INTO Airflow_testing1 SELECT * FROM [dbo].[Table1]
[2019-03-06 21:09:34,725] logging_mixin.py:95 INFO - [2019-03-06 21:09:34,722] jobs.py:2527 INFO - Task exited with return code 0
airflow airflow-scheduler
airflow airflow-scheduler
edited Mar 7 at 4:55
y2k-shubham
1,34911131
1,34911131
asked Mar 7 at 3:19
user1049198user1049198
62
62
Are you 100% sure that your query behaves correctly ,especially MSSQL's []-SquareBrackets syntax? And did you try it with setting the operators argumentautocommit=True
and the statements actually get's committed?
– dorvak
Mar 7 at 8:48
add a comment |
Are you 100% sure that your query behaves correctly ,especially MSSQL's []-SquareBrackets syntax? And did you try it with setting the operators argumentautocommit=True
and the statements actually get's committed?
– dorvak
Mar 7 at 8:48
Are you 100% sure that your query behaves correctly ,especially MSSQL's []-SquareBrackets syntax? And did you try it with setting the operators argument
autocommit=True
and the statements actually get's committed?– dorvak
Mar 7 at 8:48
Are you 100% sure that your query behaves correctly ,especially MSSQL's []-SquareBrackets syntax? And did you try it with setting the operators argument
autocommit=True
and the statements actually get's committed?– dorvak
Mar 7 at 8:48
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55035545%2fmssqloperater-not-executing-the-sql-statements-apache-airflow-dag%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55035545%2fmssqloperater-not-executing-the-sql-statements-apache-airflow-dag%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Are you 100% sure that your query behaves correctly ,especially MSSQL's []-SquareBrackets syntax? And did you try it with setting the operators argument
autocommit=True
and the statements actually get's committed?– dorvak
Mar 7 at 8:48