Debugging Spark In HDInsight

Sajib Mahmood gives various methods for debugging Spark applications running on an HDInsight cluster:

Spark Application Master

To access Spark UI for the running application and get more detailed information on its execution use the Application Master link and navigate through different tabs containing more information on jobs, stages, executors and so on.

These methods also apply for on-prem Spark clusters, although the resource locations might be a little different.

Related Posts

When tempdb Spills Attack

Josh Darnell ran into a problem with a SQL Agent job: One of my colleagues reached out to me recently about a production issue where a SQL Server Agent job had failed with this error message: Msg 1105, Level 17, State 2, Line 15Could not allocate space for object ‘dbo.SORT temporary run storage: 140737513062400’ in […]

Read More

Keeping S3 and Blob Storage in Sync

Sheldon Hull shares with us a technique to keep an S3 bucket in sync with an Azure Blob Storage blob: Moving data between two cloud providers can be painful, and require more provider scripting if doing api calls. For this, you can benefit from a tool that abstracts the calls into a seamless synchronization tool. […]

Read More

Categories

December 2016
MTWTFSS
« Nov Jan »
 1234
567891011
12131415161718
19202122232425
262728293031